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OBJECTIVE 


This  document  defines  the  scope  of,  and  manpower  requirements  for,  the 
Test  Design  and  Evaluation  effort  to  be  performed  by  the  Systems  Analysis 
Directorate  of  the  US  Army  Armament  Command  (ARMCOM)  for  implementation 
of  the  Single  Integrated  Development  Test  Policy  in  coordination  with  the 
US  Army  Test  and  Evaluation  Command  (TECOM) . 

INTRODUCTION 

One  of  the  recommendations  of  the  Army  Materiel  Acquisition  Review 
Committee  (AMARC)  was  to  eliminate  redundancy  in  the  development  testing 
process  and  to  integrate  independent  tests  that  were,  heretofore,  poorly 
coordinated.  To  implement  these  recommendations,  the  Army  Materiel  Command 
(AMC)  has  issued  a new  policy  (Appendix  A)  initiating  a Single  Integrated 
Development  Test  Cycle  (SIDTC) . This  policy  directs  the  following  actions: 

1.  Development  testing  shall  be  structured  as  an  integrated  test 
cycle  to  assure  that  the  contractor,  developer,  evaluator,  and  tester 
work  together  to  maximize  the  use  of  test  data  and  thereby  minimize  test 
cost. 

2.  A group  separate  from  both  the  developer  and  the  tester  shall 
perform  an  independent  evaluation  of  test  results  to  insure  objectivity 
in  the  analysis. 

Subsequent  AMC  guidance  directed  that  the  responsible  developer 
establish  a Test  Integrated  Working  Group  (TIWG)  for  each  "major"  and 
"designated  non-major"  development  effort  to  effect  the  necessary  coordina- 
tion. For  all  other  systems  or  family  of  systems,  a TIWG  may  be  assembled 
for  the  purpose  of  guiding  the  testing  aspects  of  each  program  in  an 
efficient  cost-effective  manner.  The  establishment  of  TIWGs  for  non-major 
systems  will  be  at  the  discretion  of  a joint  Materiel  Developer /TRADOC 
working  group.  One  of  the  significant  products  of  the  TIWG  effort  will 
be  a set  of  test  plans  for  data  acquisition.  Those  test  plans  will  insure 
that  adequate  data  are  acquired  to  permit  complete  characterization  of 
system  performance. 

As  part  of  this  analysis, the  Army  Materiel  Systems  Analysis  Agency 
(AMSAA)  has  been  designated  as  the  independent  evaluator  for  major  and 
designated  non-major  or  special  items.  For  all  other  systems,  TECOM, 
in  coordination  with  the  Major  Subordinate  Command  (MSC)  "Red  Team",  is 
given  the  independent  evaluator  responsibility.  The  decisions  reached  by 
the  TIWG  are  derived  in  part  from  the  Red  Team  evaluation  and  other  Red 
Team/Blue  Team  analyses. 

Within  ARMCOM,  the  Research,  Development  and  Engineering  Directorate 
(AMSAR-RD)  has  been  assigned  responsibility  to  implement  the  new  SIDTC 
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Policy,  and  the  Systems  Analysis  Directorate  (AMSAR-SA)  has  been  assigned 
responsibility  for  the  Red  Team  function^-.  The  Red  Team  will  work  together 
with  the  Blue  Team  during  certain  phases  of  the  test  cycle  and  independently 
of  the  Blue  Team  during  other  phases. 

The  Red/Blue  Team  concept  has  been  used  for  several  years.  The  Blue 
Team  is  the  organization  responsible  for  a given  course  of  action  (in  this 
case,  the  developer).  The  Red  Team  is  a small  group  of  analysts  or 
specialists  who  provide  an  independent  evaluation  of  the  course  of  action 
recommended  or  taken  by  the  Blue  Team.  The  AMC  concept  expands  the  Red 
Team  activities  further  to  include  actual  participation  in  the  test  design. 
This  paper  develops  (1)  the  responsibilities  and  relationships  of  the  Red 
Team  in  the  SIDTC  in  coordination  with  TECOM  and  (2)  the  workload  expected. 

FUNCTIONS  OF  THE  RED  TEAM 

The  Red  Team  will  be  actively  involved  throughout  the  development  test 
cycle  for  all  non-major  items  which  are  its  responsibility.  This  will  in- 
clude the  review  of  letters  of  agreement  and  requirements  documents,  partici- 
pation in  test  design,  monitoring  of  test,  evaluation  of  test  results,  and 
presentation  of  the  independent  overall  evaluation.  The  TECOM  task  team 
will  integrate  the  TECOM  efforts  for  planning,  conducting,  and  analyzing 
the  test  and  will  also  coordinate  the  Red  Team  participation.  The  Red 
Team  will  participate  in  all  TIWGs  for  non-major  items.  Specific  analyses 
will  be  delegated  to  the  Red  Team,  particularly  in  the  area  of  cost-effective- 
ness, risk  analysis,  and  other  systems  analysis  for  which  the  Red  Team  has 
responsibility/capability . 

The  Red  Team  functions  keyed  to  phases  of  the  life  cycle. are  described 
in  Appendix  B.  The  initial  major  task  of  the  Red  Team  will  be  the  prepara- 
tion of  the  Independent  Evaluation  Plan  (IEP)  which  is  included  in  the 
developer's  Coordinated  Test  Plan  (CTP) . The  IEP  and  the  subsequent  Inde- 
pendent Overall  Evaluation  (IOE)  will  be  joint  efforts  of  the  Red  Team 
and  TECOM  with  the  latter  being  primarily  conducted  by  the  Red  Team. 

Independent  Evaluation  Plans. 

IEPs  are  the  master  plans  detailing  action  for  evaluation  of  the 
system's  technical  and  operational  effectiveness.  The  IEP  Includes  issues 
for  testing,  identification  of  data  sources,  test  descriptions,  and  the 
approach  to  evaluation  and  reporting.  The  IEP  provides  the  basis  for  formu- 
lating the  overall  test  design.  The  IEP  has  eight  major  sections.  The 
title  of  each  section  and  the  ratio  of  the  ARMCQM  Red  Team  to  TECOM  workload/ 
responsibility  for  each  section  are  as  follows: 


Memorandum  of  Understanding,  AMSAR-SA,  subject:  Red  Team  Function  in  the 

Single  Integrated  Development  Test  Cycle,  dated  18  August  1975. 
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Section 


Red  Team/TECOM 


4.  Other  Issues 

5.  Alternatives 


6.  Evaluation  Criteria 

7.  Evaluation  Methodology 

8.  Constraints 


1.  Introduction 


2.  Objective  of  Evaluation 

3.  Critical  Issues 


50/50 

75/25 

75/25 

90/10 

80/20 

40/60 

75/25 

75/25 


Appendix  C describes  the  basic  outline  and  contents  of  each  section. 
Independent  Overall  Evaluation. 

The  Independent  Overall  Evaluation  will  be  conducted  in  accordance 
with  Section  7 of  the  IEP.  System  performance  will  be  assessed  against 
established,  coordinated, and  TRADOC- approved  scenarios  where  applicable. 
Cost/Benefit  Analysis  will  be  made  to  determine  the  desirability  of 
correcting  deficiencies  and  shortcomings  and/or  continued  development. 
Logistic  and  training  implications  will  be  addressed  and  their  impact 
on  life-cycle  cost  indicated.  For  life-cycle  phase  where  evaluations 
are  required,  the  analysis  will  include: 

1.  Cost  effectiveness 

2.  Survivability 

3.  Reliability,  Availability,  Maintainability  (RAM) 

4.  Risk  (cost , schedule,  technical) 

5.  Operational  readiness  (Logistics) 

6.  Producibility 

7.  Contractor  Constraints/Alternatives 

8.  Interaction  with  related  systems 

9.  Life  cycle  considerations 

10.  Phasing  out  old  - introducing  new 

WORKLOAD  FORECAST 

At  present,  eighteen  systems  under  ARMC0M  are  designated  for  independent 
evaluation  by  AMSAA.  They  are: 


System 


Designation 


1.  XM198-155mm,  Medium,  Towed  Howitzer 

2.  XM712-155mm  Cannon-Launched  Guided  Projectile 

3.  SAWS- (Squad  Automatic  Weapon  System) 

4.  MICV  FPW-MICV  Firing  Port  Weapon 

5.  XM58-Rapidly  Emplaced  Minefield  Marking  System 

6.  XM128-Ground  Vehicle  Dispersed  Mine  System 

7.  XM204-105mm  Soft  Recoil,  Lighti  Towed  Howitzer 


Non-major 
Non-major 
(REMMS)  Non-major 
Non-major 
Non-major 


Maj  or 
Major 


7 


8.  XM224  60mm  Lightweight  Company  Mortar  System 

9.  XM587E2  ETSQ  Fuze 

10.  XM650  8-Inch  HE,  RA,  Projectile 

11.  XM692E1  155mm  AP  Projectile 

12.  XM711  8-Inch  Extended  Range  HE  Projectile 

13.  XM718  155mm  Antitank  Projectile 

14.  XM753  8-Inch  Improved  Nuclear  Projectile 

15.  M56-Mine  Dispersing  Subsystem  (PIP) 

16.  M60A1E3  105mm  Tank,  F.T.  (M116,  M140,  M150  Gunmounts) 

17.  M60A2  152mm  Tank  Gun/Launcher  Mount 

18.  M110E2  Improved  8-Inch  SP  Howitzer 


Non-ma j or 
Non-major 
Non-major 
Non-major 
Non-major 
Non-major 
Non-major 
Non-major 
Non-major 
Non-major 
Non-major 


The  Systems  Analysis  Directorate  will  function  as  part  of  the  Blue  Team 
in  support  of  the  developer  for  the  above  items. 


Thirty-three  non-major  ARMCOM  systems  have  been  submitted  by  the 
subordinate  arsenals  for  the  establishment  of  TIWGs.  These  systems  will 
come  under  the  Red  Team  responsibility.  The  list  of  these  systems  for 
each  arsenal  follows: 


Edgewood 

1.  Projectile,  155mm,  Lethal  Binary,  GB2,  XM687 

2.  Projectile,  8-Inch,  Binary,  VX2,  XM736 

3.  Document  Destroyer  Emergency,  Incendiary  XM4 

4.  Rocket,  66mm,  TACTICAL,  CS,  XM96 

5.  Projectile,  64mm:  Riot  Control,  Kinetic  Energy,  XM743 

6.  Projectile,  64mm:  Riot  Control,  CS2,  XM742 

7.  Armored  Vehicle  Protective  Smoke  Screen 

8.  Detector  Kit,  Chemical  Agent:  XM256 

9.  Paper,  Chemical  Agent  Detector:  XM9 

10»  Modular  Collective  Protective  Equipment 

11.  Mask,  Chemical-Biological:  Multipurpose,  XM29/XM30 

12.  Alarm,  Bio  Agt,  Auto:  Chemiluminescence  XM19 

13.  Alarm,  Chemical  Agent  Automatic:  Passive  Lopair  XM21 

14.  Projectile,  105mm  Screening  Smoke,  WP  XM761 

15.  Rocket,  Smoke,  2. 75 -Inch  Screening,  WP,  XM129 

16.  Projectile,  155mm:  Smoke,  HC,  M116E2  (PIP) 

17.  Decontamination  Apparatus,  M12A1  (PIP) 

18.  Cartridge,  Smoke,  AX:  105mm  M84A1  (PIP) 

• 

Picatinny 

1.  Cartridge,  105mm,  XM710 

2.  Cartridge,  40mm,  Training,  XM- 

3.  Cartridges,  40mm,  Signal 

4.  Target  Marking  System  (XM182  Flare) 

5.  Projectile,  155mm,  HE,  XM708 

6.  Cartridge,  105mm,  HEAT-T,  XM622 
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7. 

Cartridge,  105mm,  APERS-T, 

XM665 

8. 

Mortar  Time  Fuze 

9. 

Artillery  Remotely  Settable  Fuze 

10. 

Firing  Device,  Demolition, 

XM122 

11. 

Ordnance  Locator 

12. 

Projectile,  155mm,  M483A1 

13. 

Stick  Propellant 

Rock  Island  < 

1.  RDGAD  (Radar  Directed  Gun  Air  Defense  System) 

2.  GLAADS  (Gun  Low  Altitude  Air  Defense  System) 

ARMCOM  has  an  additional  30-40  systems  under  development  (XM-designation) 
and  over  150  fielded  items  which  could  be  added  to  the  list  in  the  future. 

TOTAL  RED  TEAM/BLUE  TEAM  RESOURCE  REQUIREMENTS 

It  is  estimated  that  38  man-months  of  effort  will  be  expended  for  each 
non-major  system  by  the  Red  Team.  This  effort  will  be  spread  over  an  average 
development  time  of  6 years  and  is  broken  out  into  each  development  phase 
as  follows: 


1. 

Conceptual  phase 

8.00  man-months 

2. 

Advanced  development 

13.50  man-months 

3. 

Engineering  development 

12.25  man-months 

4. 

LRIP 

TOTAL 

4.25  man-months 
38.00  man-months 

The  major  activities  of  each  phase  and  breakout  of  the  estimated  man-months 
of  .effort  is  shown  in  Appendix  B.  Based  on  an  average  of  35  non-major  systems 
under  the  Red  Team  responsibility  and  38  man-months  of  effort  for  each  system 
over  a six  year  period,  the  annual  Red  Team  effort  will  require  18.75  man- 
years  of  effort. 

(35  systems  X 38  man-months  7 6 years  7 12  man-months/man-year  ■ 18.47  man-yrs/yr) 

Assuming  that  the  Red  Team-to-TECOM  workload  ratio  will  be  approximately 
75/25,  TECOM  will  require  (18.47  7 3)  6.15  man-years  of  effort  to  perform 
the  TECOM  mission  responsibilities  for  ARMCOM  systems.  This  totals  to  a 
combined  24.6  man-years  of  effort  per  year  for  an  average  of  35  non-major 
systems  or  .70  man-years  per  system  per  year.  This  is  compared  to  AMSAA's 
(132  man-years/58  systems)  2.28  man-years  of  effort  forecasted  for  each 
major  system  per  year. 

In  addition  to  the  Red  Team  function,  the  Systems  Analysis  Directorate 
will  also  function  in  the  Blue  Team  role  to  support  the  developer  on  major 
systems  and  designated  non-major  systems.  There  are  presently  18  ARMCOM 
systems  in  this  category,  The  Systems  Analysis  Red/Blue  Team  functions 
in  support  of  the  developer  Include: 
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1.  Establishing  requirements  for  a materiel  performance  technical 
data  base  for  use  in  the  overall  test  and  evaluation  process  and  validating 
technical  information  and  data  used  in  the  evaluation  process. 

2.  Performing  parametric  analyses  and  trade-off  studies  to  support 
test  design,  test  planning,  and  test  assessment  efforts. 

3.  Preparing  risk  analysis  and  risk  statements  based  on  ahalysis  of 
test  data. 

4.  Developing  mathematical  and  simulation  models  which  will  permit 
the  study  of  the  ARMCOM  view  of  the  system  under  a very  broad  range  of 
potential  environmental  and  field  scenario  conditions. 

5.  Developing  system  cost  and  effectiveness  analysis  methods  and 
techniques  for  application  to  estimating  total  system  cost. 

6.  Performing  and  evaluating  cost  and  system  effectiveness  studies 
as  required  by  higher  headquarters  or  by  internal  operations  of  ARMCOM. 

7.  Comparing  and  evaluating  alternative  test  designs  and  test  plans 
in  order  to  achieve  maximum  effectiveness  at  the  most  economical  costs. 

These  Blue  Team  functions  of  the  Systems  Analysis  Directorate  (Red 
Team)  on  major  and  designated  non-major  systems  will  require  approximately 
the  same  resources  per  system  as  the  non-major  systems  under  the  Red 
Team.  Based  on  an  average  of  15  major /designated  non-major  systems  being 
supported  by  the  Blue  Team,  this  will  require  an  additional  8 man-years 
of  effort.  (15  systems  X 38  man-months  7 6 years  7 12  man-months/man-year  - 
7.9  man-year s/ year ) . 

SYSTEMS  ANALYSIS  DIRECTORATE  RESOURCE  REQUIREMENTS 

The  ARMCOM  manpower  requirements  analysis  assumes  that  for  the  Red 
Team  function  the  workload  breakdown  will  be  similar  to  AMSAA's  and, 
furthermore,  will  require  effort  from  other  directorates  within  ARMCOM. 

The  AMSAA  manpower  requirements  analysis  for  major,  designated  and 
selected  non-major  systems  indicated  a workload  breakdown  covering  ten 
subfunctions^ . These  subfunctions  are  shown  below  with  the  percentage 
of  manpower  requirements  allocated  to  each  subfunction: 

TD&E  Subfunction  Manpower  (%) 


1. 

Test  Design 

7.8 

2. 

Monitor  Concurrent  Analysis 

8.6 

3. 

Data  Analysis 

10.5 

4. 

Evaluation 

13.6 

5. 

Administrative 

2.3 

6. 

RAM  Analyses 

12.7 

2 

ECOM  Red  Team  Role  in  the  Single  Integrated  Development  Test  Cycle. 
Systems  Analysis  Office,  US  Army  Electronics  Manual,  Ft.  Monmouth,  NJ. 
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TD&E  Subfunction 


Manpower  (%) 


7. 

Logistic  Analysis 

10.6 

8. 

Economic  Analysis 

3.9 

9. 

Survivability  Analysis 

3.0 

10. 

Systems  Analysis 

27.0 

TOTAL 

100.0 

For  subfunction  06,  it  is  assumed  that  RAM  Analyses  will  require  equal 
amounts  of  effort  from  the  Systems  Analysis  Directorate  and  the  Product 
Assurance  Directorate.  For  subfunction  01 , it  is  expected  that  90%  of 
the  Logistics  Analysis  effort  will  stem  from  the  Systems  Analysis  Directorate 
while  10%  will  be  provided  by  the  Maintenance  Directorate.  For  subfunction 
OS,  the  Economic  Analysis  workload  will  be  shared  equally  between  the 
Systems  Analysis  Directorate  and  the  Cost  Analysis  Division  of  the  Comp- 
troller. The  support  of  these  other  ARMCOM  activities  will  be  in  addition 
to  their  other  responsibilities  (Blue  Team)  realted  to  the  TD&E  function. 

The  impact  of  this  requirement  on  the  ARMCOM  elements  is  as  follows: 

Red  Team  Function  effort:  18.47  man-years /year 

Product  Assurance  (subfunction  06) : 

0.5  X .127  X 18.47  - 1.17 

Maintenance  Engineering  (subfunction  01): 

0.1  X .106  X 18.47  - .20 

Cost  Analysis  - Comptroller  (subfunction  OS): 

0.5  X 0.039  X 18.47  - .36 
Systems  Analysis  Directorate: 

18.47  - (1.17  + .20  + .36)  ■ 16.74  man-years 
Blue  Team  Function  effort  ■ 7.9  man-years 

Total  requirement  for  Systems  Analysis  Directorate  ® 24.64  man-years /year 

The  conclusion  drawn  is  that  25  man-years  of  dedicated  professional 
personnel  are  required  in  the  Systems  Analysis  Directorate  for  Test  Design 
and  Evaluation  on  a full  time  basis.  These  personnel  exclude  those  now 
serving  to  fulfill  the  existing  mission  requirement  of  the  directorate. 

Two  to  three  professionals  from  other  ARMCOM  activities  are  required  to 
support  the  total  Red  Team  TD&E  requirement, 


11 


Next  page  is  blank. 


— 


APPENDIX  A 


SINGLE  INTEGRATED  DEVELOPMENT  TEST  CYCLE  IMPLEMENTATION  LETTER 


13 


Next  page  is  blank. 


DEPARTMENT  OF  THE  ARMY 

HCADQUARTGRS  UNITCO  OTATCS  ARMY  MATCHICL  COMMAND 
5001  CISCNHOWCR  AVE.,  ALEXANDRIA,  VA -XO.KK  22333 


AMCRD-U 

SUBJECT:  Single  Integrated  Development  Test  <?yde  Tolicy 

/' 


81  jan^5?5 


SEE  DISTRIBUTION 
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a.  AR  1000-1,  Basic  Policies  for  Systenjs  Acquisition  by  the. Department  . 

of  the  Army , 5 November  1974 . 

• • 

b.  AMCQA  letter,  subject:  Development  Test  Policy , 26  November  1974 . 

2.  Reference  la  establishes  the  policy  that  the  contractor  and  AMC  develop- 
mental testing  should  be  integrated  into  one  cycle.  The  AMC  plan  of  action 
for  implementing  this  policy  has  been  approved  by  CG,  AMC.  Pending  pub- 
lication of  appropriate  AMC  regulations,  interim  guidance  for  implementation 
is  contained  in  Inclosures  1 through  4.  Reference  lb  is  hereby  rescinded. 

The  integrated  test  cycle  plan  of  action  incorporates  the  test  policy  contained 
therein . Inclosure  2 to  reference  lb  (IjlDAT  study}  may  be  retained  for 
information . 

3.  Implementation  of  the  Single  Integrated  Development  Test  Cycle  policy  will 
be  accomplished  as  follows: 

a.  All  new  programs  will  implement  the  plan  of  action  at  program  inception. 

b.  Within  current  funding  and  scheduling  constraints,  all  current  con- 
tracts and  on-going  programs  will  be  restructured  with  the  objective  of 
implementing  the  new  integrated  t6st  concept  to  the  maximum  extent  possible. 
Developers  will  provide  AMSAA  and  TECOM  appropriate  contractor  test  data 
that  are  available  within  current  contracts.  TECOM,  in  coordination  with 
AMSAA,  will,  to  the  extent  possible,  reduce  the  scope  of  currently  planned 
DT  II  and  DT  HI  by  making  maximum  utilization  of  the  contractor  test  data. 

• I 1 • 
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Single  Integrated  Development  Test  Cycle  Policy 


c.  AMSAA  phase-in  plan  (Inclosurc  3)  for  accomplishing  test  design 
and/or  evaluation  is  provided  for  planning  purposes.  AMSAA's  functions 
and  responsibilities  take  effect  immediately  for  all  major,  designated  non- 
major and  other  selected  systems.  Phase-in  is  dependent  on  allocation 
and  accomplished  on  a casc-by-caso  basis.  Somo  systems  arc  currently 
undergoing  tests,  thus  precluding  test  design  for  the  on-going  phase  of 
the  development  cycle.  Developers. should  coordinate  with  AMSAA  to  arrange 
for  a logical  and  appropriate  phase-in  point.  However,  AMSAA  and  developers 
should  make  every  effort  tophase-in  all  systems  as  soon  as  possible.  Toi3) 


-in  of  TECOM  and  the 
jvaluators,  will  be 


those  systems  that  AMSAA  does  not  evaluate',  phas 
MSC  "Red  T-eam"  in  the  role  of  test  designers  and 
developed  by  TECOM  in  coordination  with  the  MSCjs.-  Developers  should 
coordinate  with  TECOM  to  arrange  for  a logical  and  appropriate  phase-in 
point . 


d.  Phase-in  for  current  on-going  programs  should  be  accomplished,  at 
the  latest,  at  the  next  major  decision  point  (IPR/ASARC/DSARC)  for  imple- 
mentation in  the  follow-on  Development  phase  (AD,  ED,  LRIP)  or  when  a 
program  is  restructured  as  a result  of  program  reorientation  or  change  in 
requirement. 

e.  Developers  will  initiate  immediate  action  to  establish  Test  Integration 
Working  Groups  (TIWG)  for  all  systems  currently  under  development  (Incl  1)  . 
Identification  of  organizational  elements  of  commands/activities  participating 
in  the  TIWG  will  be  provided  to  this  headquarters,  ATTN:  AMCRD-U,  as  soon 
as  possible  but  not  later  than  the  initial  scheduled  status  report  date  listed 

in  3f  below . 


f.  Progress  on  implementation  will  be  reported  as  follows: 

(1)  Beginning  1 March  1975,  project  managers  presenting  RECAPS  to  this 
hoadquarters  will  includo  a status  report  as  part  of  their  RECAPS . 

(2)  Beginning  1 April  1975,  major  subordinate  commands  will  include  a 
status  report  as  part  of  the  RDfjE,  Quarterly  Review  and  Analysis, 


(3)  Beginning  March  1975,  TECOM  will  include  a status  roport  covering 
reduction  in  currently  planned  DT  II  and  DT  III  in  the  TECOM  bimonthly  Test 
Problem  Briefings  to  the  CG,  AMC. 

(4)  Beginning  March  1975,  AMSAA  will  incluae  a status  report  covering 
their  test  design  and  evaluation  mission  in  the  AMSAA  monthly  Kay  Efforts 
Roport  to  HQ  AMG . 
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• » *# 

4.  It  is  recognized  that  the  AMC  plan  of  action  docs  not  adequately  address 

the  interface  with  the  operational  tester.  By  separate  correspondence  this 
headquarters  will  recommend  to  DA  that  AMC,  TRADOC  an  OTEA  form  a 
working  group  to  expand  the  AMC  plan  of  action  and  restructure  the 
Coordinated  Test  Program.  I 

5.  The  objective  of  this  policy  is  to  maximize  the  use  of  contractor  testsf 
thereby  reducing  testing  required  by  AMC*.  reduce  costs  and  preclude  dup- 
licate testing.  Commanders  and  project  managers  are  requested  to  make' 
maximum  dissemination  of  the  inclosed  policies. 


4'  Incls 

1*.  DT  Policy 

2.  Responsibilities  of  AMC 

elements 

3.  AMSAA  Phase-In  Plan 

4.  A DT  Test  Cycle  Scenario 

DISTRIBUTION: 

A fi  B 
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SINGLE  INTEGRATED  DEVELOPMENT  TEST  POLICY 


1.  Development  testing  shall  be  structured  as  an  integrated  test  cycle 
to  assure  that  the  contractor,  developer,  AMSAA  and  TECCM  interact  to 
minimize  test  cost  and  maximize  the  use  of  test  data.  DT  I,  DT  II  and 
DT  III  tests  include  those  tests  performed  by  the  contractor,  AMC  major 
subordinate  commands,  project  managers  and  TECCM  that  are  utilized  for 
the  overall  system  evaluation.  The  Government  portion  of  development 
testing,  keyed  to  each  materiel  acquisition  phase,  will  be  so  structured 
as  to  utilize  validated  contractor  generated  data  to  avoid  duplicate 
testing.  Inherent  to  this  policy  is  that  successful  development  decision 
making  lies  in  independent  evaluation  of  valid  data  rather  than  the  inde- 
pendent conduct  of  tests. 

2.  The  integrated  development  test  policy  demands  a close  and  continuous 
interface  between  the  developer,  AMSAA,  TECCM  and  the  contractor.  Close 
coordination  with  the  operational  tester  must  also  be  effected  to  mini- 
mize time  and  cost  and  to  preclude  duplication  between  development  testing 
and  operational  testing.  To  facilitate  the  integration  of  test  require- 
ments and  to  speed  the  coordination  process,  the  developer  will  establish 
and  chair  a Test  Integration  Working  Group  (TIWG) . This  group  will  be 
formally  chartered  and  consist  of  members  having  authority  to  act  for 
their  respective  commands/activities.  Membership  should  include  repre- 
sentatives of  the  Combat  Developer,  Logistician,  Operational  Tester, 

AMSAA,  TECCM  and  where  appropriate,  the  contractor.  The  primary  purpose 
of  the  TIWG  is  to: 

a.  Assist  the  developer  in  the  preparation  of  Coordinated  Test  Program 
(CTP) . 

b.  Monitor  the  test  program  progress. 

c.  Update  the  CTP  as  required. 

3.  The  Coordinated  Test  Program  (CTP)  will  be  utilized  as  the  key  management 
tool  for  control  of  the  single  integrated  Contractor/Developer/Tester 
development  test  cycle. 

• 

4.  Test  design,  planning,  programming,  budgeting  and  execution  must  be 
a coordinated  activity  among  the  contractor,  materiel  developer,  tester 
(TECCM)  and  evaluator  to  provide  for  a single,  well  integrated,  develop- 
ment test  cycle  to  minimize  costs,  maximize  utilization  of  test  data 
and  standardize  definitions  and  criteria. 

5.  Test  designers,  through  early  planning,  must  coordinate  test  design 
efforts  with  the  objective  of  requiring  a minimum  number  of  tests  and 
test  items  to  obtain  the  necessary  data  for  evaluations.  Number  of 
prototypes,  sample  size,  hours  of  testing  and  other  test  design  criteria 
will  be  established  on  the  basis  of  analyses  of  cost,  time  and  risks  in 
relation  to  the  decision  process. 


INCL  1 
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6.  Models,  simulators,  statistical  design  of  experiments,  systems 
analysis,  engineering  analysis,  operations  analysis,  and  data  bases  in 
lieu  of,  or  to  supplement  tests  will  be  used  to  strengthen  evaluation 
and  reduce  test  costs. 

7.  Contracts  will  be  structured  to  assure  that  development  hardware  is 
not  submitted  for  Government  testing  until  the  contractor  has  demonstrated 
that  contract  requirements  have  essentially  been  met  and  the  developer 

has  a high  level  of  confidence  that  Government  testing  will  be  successfully 
concluded.  The  objective  is  to  obviate  the  conduct  of  costly  and  time 
consuming  Government  testing  of  prematurely  developed  hardware  that  has 
little  chance  for  success. 

8.  Requests  for  Proposals  (RFPs)  will  include  the  test  requirements  to 
which  the  system  will  be  tested  during  DT.  Executed  contracts  will  in- 
clude the  tests  to  be  conducted  by  the  contractor,  by  the  Government  or 
jointly  by  contractor  and  Government. 

9.  Insofar  as  possible,  contracts  will  require  that  testing  to  demonstrate 
compliance  with  contract  hardware  requirements  be  conducted  by  the  con- 
tractor. It  is  recognized  that  for  conduct  of  certain  tests,  particularly 
those  involving  environmental  and  terrain  conditions,  contractors  may  not 
have  the  required  facilities.  For  these  tests  (within  the  command  policy 
of  maximizing  the  utilization  of  TECCM  facilities  as  expressed  in  19  Apr 

73  AMCEMA  letter,  subject:  Test  and  Evaluation  Role  of  USATECCM)  the 
following  alternatives  apply: 

a.  Where  practicable,  developers  will  arrange  with  TECOM  to  offer 
TECCM  facilities  to  the  contractor  for  conducting  the  tests  using 
contractor  personnel.  When  contractor  conducted  testing  utilizing  TECCM 
facilities  is  not  feasible,  these  tests  will  be  conducted  by  TECCM  after 
developer/TECCM  coordination.  TECCM  will  develop  viable  procedures  to 
maximize  the  use  of  TECCM  facilities  by  AMC  development  contractors. 

b.  Where  practicable  and  where  developer  test  facilities  are  already 
available,  arrangements  for  the  contractor  to  conduct  his  tests  using 
these  facilities  will  be  arranged  in  accordance  with  the  policy  expressed 
in  9a  above. 

10.  When  the  DD  Form  250  is  used  to  move  development  hardware  from  the 
contractor's  facility  for  Government  testing,  it  shall  be  appropriately 
annotated  to  show  that  the  materiel  is  to  be  used  for  test  purposes  only, 
and  not  to  denote  acceptance  of  hardware  as  meeting  contract  requirements. 
Hardware  acceptance  will  be  made  only  when  the  contracting  officer  has 
determined  that  contract  compliance  has  been  achieved. 

11.  Prior  to  proposal  solicitation,  each  developer  will  conduct  a formal 
review  for  traceability  of  requirements  from  the  system  specification/ 
work  statement  back  to  the  approved  LQA,  ROC  or  LR.  The  purpose  is  to 
obtain  an  objective  review  and  consensus  that  LQA,  ROC  or  LR  requirements 
have  been  translated  to  technical  requirements  and  that,  if  met,  will 
meet  the  LQA,  ROC  or  LR  objectives.  For  procurement  review  board  actions, 
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the  results  of  the  traceability  review  shall  be  presented  to  the  board. 
User  participation  in  the  traceability  review  is  encouraged. 


Responsibilities  of  the  Developer  (MSC/PM) 
AMSAA  and  THCQM 


1.  Responsibilities  of  the  Developer  (MSC/FM) : 

a.  Preparation  of  the  Coordinated  Test  Program  (CTP)  in  coordination 
with  the  Combat  Developer,  Operational  Tester,  Logistician,  AMSAA  and 
TECCM  and  with  appropriate  contractor  participation. 

b.  Management  of  the  Development  Test  portion  of  the  Coordinated 
Test  Program  (CTP). 

2.  Responsibilities  of  AMSAA: 

a.  Preparation  of  the  Independent  Evaluation  Plan  for  DT  I,  DT  II, 
and  DT  III  for  major,  designated  non-major  and  other  selected  systems; 
also  to  overview  the  other  systems  on  a sampling  basis.  This  plan  de- 
tails actions  for  acquiring  sound  test  data  responsive  to  the  decision 
process  and  provides  the  basis  for  formulation  of  the  overall  test 
design.  The  Plan  is  coordinated  with  the  Developer  (MSC/PM)  and  TECCM 
and  is  provided  as  input  to  the  Coordinated  Test  Program  (CTP) . 

b.  Preparation  of  the  overall  test  design  for  DT  I,  ET  II,  and  DT  III 
for  major,  designated  non-major  systems  and  other  selected  systems.  The 
test  design  serves  as  the  basis  for  determining  those  tests  that  will  be 
performed  by  the  contractor,  AMC  major  subordinate  conmands,  project 
managers  and  TECCM  which  will  be  used  for  the  overall  system  evaluation. 
The  test  design  is  coordinated  with  the  Developer  and  TECCM,  and  pro- 
vided as  input  to  the  Coordinated  Test  Program  (CTP) . 

c.  Preparation  of  the  Independent  Overall  Evaluation  of  major  and 
designated  non-major  systems  and  other  selected  systems  for  presentation 
to  the  Developer  and  CG  AMC. 

3.  Responsibilities  of  TECCM: 

a.  For  systems  which  AMSAA  is  not  the  evaluator: 

(1)  Preparation  of  the  Independent  Evaluation  Plan  for  DT  I,  DT  II 
and  DT  III.  This  plan  details  actions  for  acquiring  sound  test  data 
responsive  to  the  decision  process  and  provides  the  basis  for  formula- 
tion of  the  overall  test  design.  The  plan  is  developed  in  coordination 
with  the  major  subordinate  command's  "Red  Team".  The  plan  is  also 
coordinated  with  the  MSC's  materiel  proponent  and  is  provided  as  input 
to  the  Coordinated  Test  Program. 

(2)  Preparation  of  the  overall  test  design.  The  test  design  serves 
as  a basis  for  determining  those  tests  that  will  be  performed  by  the 
contractor,  AMC  major  subordinate  commands,  project  managers  and  TECCM, 
which  will  be  used  for  the  overall  evaluation.  The  test  design  is  co- 
ordinated with  the  MSC's  materiel  proponent  and  provided  as  input  to 
the  Coordinated  Test  Program. 


INCL  2 
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(3)  Preparation  of  the  Independent  Overall  Evaluation  in  coordination 
with  the  MSC  "Red  Team"  for  presentation  to  the  Developer  and  CG  AMC. 

b.  For  all  systems,  TECCM  will  plan,  conduct  and  prepare  analysis 
of  government  validation  tests,  when  required.  (Advanced  Development 
Verification  Test  - Government,  Prototype  Qualification  Test  - Government, 
and  Production  Validation  Test  - Government).  The  MSC's  materiel  propo- 
nent and  AMSAA  will  also  conduct  their  own  independent  analyses  of  these 
test  results,  as  required.  TECCM  will  provide  test  reports  and  analysis 
of  test  results  to  the  MSC's  materiel  proponent  and  AMSAA  for  systems 
evaluation. 
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AMSAA  PHASE-IN  PLAN 


SAM-D  DT  II  Test  Analysis  and  Overall  Evaluation  Jan  75 
TACFIRE  DT  III  Test  Design  Jan  75 
DSCS  DT  II  Test  Design  Jan  75 
MICV  FPW  DT  II  Test  Design  Jan  75 
Per.  Armor  DT  II  Test  Design  Jan  75 
TACSATCOM  DT  II  Test  Design  Jan  75 
AN/TPQ-36  DT  I Test  Analysis  and  Overall  Evaluation  Feb  75 
SAW  DT  II  Test  Analysis  and  Overall  Evaluation  Feb  75 
UET  DT  II  Test  Design  Apr  75 
CLGP  DT  I Test  Analysis  and  Overall  Evaluation  May  75 
XM198  DT  II  Test  Analysis  and  Overall  Evaluation  May  75 
DRONES  DT  I Test  Design  May  75 
AN/TPQ-37  DT  II  Test  Analysis  and  Overall  Evaluation  Jun  75 
XM  1 DT  I Test  Design  Jul  75 
SHORAD  DT  I Test  Design  Jul  75 
AH1Q  DT  II  Test  Analysis  and  Overall  Evaluation  Jul  75 
XM204  DT  II  Test  Analysis  and  Overall  Evaluation  Jul  75 
STINGER  DT  II  Test  Analysis  and  Overall  Evaluation  Oct  75 
XM224  DT  II  Test  Analysis  and  Overall  Evaluation  Oct  75 
MICV  DT  II  Test  Analysis  and  Overall  Evaluation  Apr  76 
8 Inch  Proj . DT  II  Test  Analysis  and  Overall  Evaluation  Jul  76 
UTTAS  DT  II  Test  Analysis  and  Overall  Evaluation  Sep  76 


INCL  3 


23 


Next  page  is  blank. 


APPENDIX  B 


l 


RED  TEAM  ACTIVITIES 
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RED  TEAM  ACTIVITIES 


Conceptual  Phase  - (8  man-months)  Man-months 

1.  Review  the  draft  Letter  of  Agreement  (LOA)  with  regard  to  the  (.5) 

following: 

a.  Confirm  or  challenge  the  need  for  the  system/subsystem. 

b.  Determine  whether  the  planned  development  is  consistent  with  the 
state-of-the-art . 

c.  Note  whether  or  not  items  which  require  testing  are  identified. 

2.  Generate  a draft  version  of  the  Independent  Evaluation  Plan  (IEP)  (4.0) 

for  the  DT  I program. 

3.  Assist  TECOM,  in  cooperation  with  the  developer,  to  generate  a (3.0) 

draft  version  of  the  Overall  Test  Design  Plan  (OTDP)  for  DT  I. 

4.  Review  the  developer’s  Coordinated  Test  Plan  (CTP) . (.5) 

Validation  or  Advanced  Development  (AD)  Phase  - (13.5  man-months) 

1.  Review  the  testing  aspects  of  the  AD  Request  for  Proposals  (RFP) . (.5) 

2.  Refine  the  IEP  for  DT  I.  (.5) 

3.  Refine  the  OTDP  for  DT  I.  (.5) 

4.  Review  developer's  updated  CTP.  (.25) 

5.  Review  the  following  test  plans  for  DT  I:  (1.5) 

a.  Engineering  Design  Test  (EDT-C/G) 

b.  Advanced  Development  Verification  Test  (ADVT-C/G) 

6.  Conduct  Independent  Evaluation  for  DT  I,  and  present  results  to  (3.0) 

Materiel  Developer  and  CG,  AMC. 

7.  Generate  the  Independent  Overall  Evaluation  (IOE)  report  for  DT  I.  (2.0) 

8.  Review  TRADOC’s  draft  ROC.  (.25) 

9.  Generate  a draft  IEP  for  DT  II.  (1.5) 

10.  Assist  TECOM,  in  cooperation  with  the  developer,  to  generate  a (1.5) 

draft  OTDP  for  DT  II. 
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Engineering  Development  (ED)  Phase  - (12.25  man-months) 

1.  Review  the  testing  aspects  of  the  ED  RFP.  (.5) 

2.  Refine  the  IEP  for  DT  II.  (.25) 

3.  Refine  the  OTDP  for  DT  II.  (.25) 

4.  Review  developer's  updated  CTP.  (.25) 

5.  Review  the  following  test  plans  for  prototype  models:  (.5  ) 

a . EDT-C/G 

b.  Prototype  Qualification  Test  (PQT-C/G) 

6.  Conduct  independent  evaluation  for  DT  II  and  present  results  to  (5.0) 

Materiel  Developer  and  CG,  AMC. 

7.  Generate  the  IOE  report  for  DT  II.  (2.0) 

8.  Review  the  test  provisions  of  the  developer's  technical  data  (.25) 

package. 

9.  Generate  a draft  IEP  for  DT  III.  (1.5) 

10.  Assist  TECOM,  in  cooperation  with  the  developer,  to  generate  a draft  (1.5) 
OTDP  for  DT  III. 

11.  Review  proposed  amendments  to  ROC.  (.25) 

Low  Rate  Initial  Production  (LRIP)  Phase  - (4.25  man-months) 

1.  Review  the  testing  aspects  of  the  LRIP  RFP.  (.5) 

2.  Refine  the  IEP  for  DT  III,  (.25) 

3.  Refine  the  OTDP  for  DT  III.  (.25) 

4.  Review  developer's  updated  CTP.  (.25) 

5.  Review  Production  Validation  Test  (PVT-C/G)  plan.  (.25) 

6.  Analyze  the  PVT-C/G  report.  (.25) 

7.  Perform  an  Independent  Overall  Evaluation  (IOE)  of  all  DT  III  tests.  (2.5) 
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DT  INDEPENDENT  EVALUATION  PLAN  FOR 


(name  of  phase  of  development) 

(Item  nomanclature) 

1.  Introduction.  Describes  the  scope  of  the  plan  and  approach  to  be 
used.  The  expected  source  of  input  data  are  described,  whether  test 
reports,  published  data,  cost/operational  effectiveness  studies  or 
others. 

2.  Objectives  of  the  Evaluation.  The  objectives  state  the  reasons  for 
making  the  evaluation.  The  objectives  are  subject  to  refinement  as  the 
plan  takes  shape.  They  should  not  be  stereotyped,  but  should  be  thoroughly 
defensible  on  the  basis  of  the  needs  of  the  program.  For  example,  the 
objective  may  be  to  evaluate  a new  systems  concept  for  Army  use  (DT  I), 

to  demonstrate  that  technical  risks  have  been  identified  and  that  com- 
ponent interface  problems  have  been  pinpointed  (DT  I) , to  determine 
readiness  for  transition  into  production  (DT  II),  to  determine  which  of 
several  competing  prototypes  has  the  best  potential  for  development  (DT 
I),  or  simply  to  determine  whether  or  not  a system  is  ready  to  proceed 
in  the  development  cycle. 

3.  Critical  Issues.  Critical  issues  are  those  identified  by  the  decision 
authority.  The  issues  will  be  limited  to  those  appropriate  to  the  specific 
phase  of  development  (i.e.,  Conceptual,  Validation,  Full-Scale  Development, 
or  Production  and  Deployment)  and  that  will  be  addressed  in  the  evaluation. 

4.  Other  Issues.  For  example,  materiel  requirements,  choice  between 
competitive  candidates,  comparison  to  standard  materiel,  design/engineering 
approaches,  or  contractual/specification  aspects.  Also  included  are 
those  issues  not  required  for  the  evaluation  but  considered  to  be 
essential  to  other  organizations. 

5.  Alternatives.  All  the  alternatives  possible  for  satisfaction  of  the 
function  to  be  performed  or  requirement  to  be  satisfied  are  considered. 

Some  can  be  discarded  immediately.  Possible  alternatives  to  adopting  a 
new  weapon  system  are:  continue  "With  the  old  system,  redesign  completely, 

eliminate  the  need  by  avoidance  of  the  particular  tactical  situation, 

buy  commercial,  contract  out,  and  others.  Alternatives  will  further  be 
discussed  in  paragraph  7d. 

6.  Evaluation  Criteria.  The  evaluation  criteria  are  not  "pass/fail" 
criteria  but  guides  for  use  in  conducting  the  Independent  evaluation  and 
for  use  in  evaluating  outcomes  to  determine  which  alternative  is  pre- 
ferable or  what  action  is  indicated  to  make  one  of  them  preferable.  The 
criteria  will  be  keyed  to  the  specific  phase  of  development  and,  when 
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appropriate,  will  reflect  performance  and  reliability  growth  predictions 
agreed  to  by  the  materiel  and  combat  developers,  even  if  different  from 
those  stated  in  the  requirement  document  (e.g.,  the  LOA  for  DT  I,  the 
ROC,  LR  or  TDR  for  DT  II,  or  the  TDP  for  DT  III). 

7.  Evaluation  Methodology.  This  section  details  the  analytic  methodology 
that  will  be  used  in  the  evaluation.  This  must  include  the  use  of  scoring 
and  weighting  factors,  mathematical  analyses,  simulations,  or  other  techni- 
ques. Only  issues  identified  in  paragraphs  3 and  4 will  be  included. 

a.  Models . A mathematical  or  logical  model  of  the  system  should  be 
devised,  which  represents  a simplified  likeness  of  the  real  situation. 

The  model  may  appear  as  a decision  tree,  one  or  more  equations,  an  itera- 
tion of  descriptive  statements  or  a combination  of  these.  In  any  case, 
the  model  must  permit  the  inclusion  of  quantitative  values  in  such  a way 
that  some  outcome  is  described  or  predicated.  A sensitivity  analysis 
should  be  accomplished  to  show  which  factors  deserve  greater  emphasis. 

b.  Threat . A paragraph  which  states  concisely  the  threat.  In 
particular,  the  existing  and  perceived  threat  that  this  system  is  expected 
to  encounter  will  be  explicitly  identified.  Consideration  should  be 
given  to  factors  such  as,  potential  vulnerability  to  known  hostile  EW 

and  CBR,  enemy  forces  and  tactics.  In  some  cases  the  total  hostile 
environment  must  be  modeled  to  permit  a thorough  evaluation  of  the  system. 

c.  Analysis  Plan.  The  plan  will  address  each  issue  stated  in 
paragraphs  3 and  4.  For  each  issue  either  a single  measure  of  effec- 
tiveness (MOE)  or  a set  of  MOEs  will  be  defined.  MOES  will  be  the  prime 
quantitative  output  of  the  DT  and  will  be  the  prime  determinant  of  what 
needs  to  be  measured,  why  it  needs  to  be  measured,  and  how  the  data  will 
be  analyzed.  The  analysis  plan  will  be  represented  by  a decision  model 
that  integrates  all  appropriate  MOEs  into  a logical  and/or  mathematical 
format  to  answer  the  specific  objectives  of  paragraph  2.  The  evaluation 
plan  must  provide  for  the  test  designers,  a priority  listing  of  variables. 
Considerations  will  be  given  to  the  most  important  variables  and  expected 
significant  interactions.  If  particular  attention  is  to  be  given  to 
Human  Factors  problems  that  also  should  be  identified  and  specific  MOEs 
for  Human  Factors  evaluations  stated,  as  appropriate.  This  paragraph 
summarizes  the  detailed  analysis  information  contained  in  annexes.  For 
simple,  low-cost  items  not  requiring  extensive  evaluation,  the  annexes  may 
be  omitted  and  the  complete  analysis  plan  presented  in  this  paragraph. 

d.  Analysis  of  Alternatives.  Procedures  for  analysis  of  each  alter- 
native identified  in  paragraph  5,  Alternatives,  will  be  presented. 

8.  Constraints . This  section  should  detail  limitations  in  the  evaluation 
due  to  time,  fiscal,  or  other  constraints.  Compromises  which  must  be  made 
and  those  which  cannot  be  tolerated  should  be  stated. 
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